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Abstract 

This study examined technology implementation practices associated with 
student learning gains. Interviews and observations were conducted with staff 
at schools where teachers using reading or mathematics software with their 
students attained above-average achievement gains and at schools where 
software-using teachers had below-average gains. The findings highlight the 
importance of school practices in the areas of principal support and teacher 
collaboration around software use and of teacher practices concerning class- 
room management and use of software-generated student performance data. 
The issues of instructional coherence and competition for instructional time 
are highlighted as challenges to software implementation. (Keywords: Tech- 
nology, implementation, software) 


O bservers of technology use in schools and classrooms have long 
noted the relatively modest use of educational technology within 
most schools and classrooms (Cuban, 2001). As the lives of students 
and teachers outside of school have evolved to include more and more use 
of technology, the situation presents a paradox. Despite decades of national, 
state, and local promotion of educational uses of technology, classroom 
practice in most schools has changed little from that of the mid-20* century. 
Recent large-scale national surveys of teacher practices with technology 
found an increase in teacher use of technology as a productivity tool sup- 
porting their own work between 2005 and 2007 but no increase in the level 
of teacher- assignment of technology-based learning activities for students 
during the same time period (Bakia, Means, Gallagher, Chen, & Jones, 

2009). Teachers and students use technology more frequently outside of 
school than they do during class time. 

Although many teachers certainly are using today’s technologies in in- 
novative ways, they remain the exception rather than the rule. In terms of 
Moore’s (1999) innovation adoption model, few learning technologies have 
managed to “cross the chasm” from adoption by technology enthusiasts and 
visionaries to acceptance by the vast majority of teachers, who are pragma- 
tists and conservatives. 

Technology adoption and implementation require not just funding 
resources but also ongoing effort. The premise underlying this paper is 
that teachers’ and school systems’ fundamental priorities concern student 
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Table 1 . Recommended School-Level Instructional Technology Practices 

Prior Research Support tor Practice 

Controlled 
Studies on 

Implementation Correlation with Correlation with Technology 

Recommendation Recommended by Technology Use Learning Outcomes Use 

Schoolwide Coherence 


Technology use integrated 
with a consistent school-wide 
instructional vision 

Barnett (2002) 

Means & Olson (1 995) 
OTA (1995) 

Means & Olson 
(1995) 


Technology aligned with local 
curriculum 

Barnett (2002) 
Ertmer (1 999) 
Sarama et at (1 998) 
Sweet et al. (2004) 



Principal demonstration ot sup- 
port for technology integration 

Brand (1997) 
Coley et al. (1 997) 
OTA (1995) 

Mann etal. (1998) 
O'Dwyer etal. 
(2004, 2005) 

Zhao etal. (2002) 



Teacher Training 


Teachers trained on concepts 
of student-centered teaching 
and 

technology integration 

Barnett (2002) 

Becker (1 994, 
2000) 

Mann etal. (1998) 
O'Dwyer etal. 
(2004, 2005) 

Zhao etal. (2002) 

eMINTS (2003) 
Wenglinsky (1 998) 

Teachers trained on implemen- 
tation of the specific software/ 
innovation 

EETI vendors 

Becker (1 994) 
Mann etal. (1998) 
U.S. Department of 
Education (2000) 

Mann etal. (1998) 

Professional development is 
ongoing, not one-time (e.g., 
mentoring or coaching) 

Brand (1997) 
Jones et al. (1 995) 
OTA (1995) 

Adelman et al. 
(2002) 

Becker (1 994) 

U.S. Department of 
Education (2000) 

Cole, Simkins, & Penuel 
(2002) 

Professional development 
involves teachers in designing 
technology-supported learning 
activities/resources 


Martin et al. (2003) 

Yamagata-Lynch 

(2003) 



Technology Access 


Computers/Internet accessible 
in regular classrooms 

Barnett (2002) 
Mann et al. (1 998) 
OTA (1995) 

Becker (2000) 

Mann etal. (1998) 

Adequate access to technology 
for all students 

Barnett (2002) 

O'Dwyer etal. 
(2004, 2005) 



Support for Technology Use 


Technical support available at 
the school 

Barnett (2002) 
Sweet et al. (2004) 

Becker (1 994) 

Hill & Reeves (2004) 
Zhao etal. (2002) 

Cole, Simkins, & Penuel 
(2002) 

Teachers collaborate around 
technology use 

Brand (1997) 

Becker (2000) 
Frank, Zhao, & 
Borman (2004) 
Means & Olson 
(1995) 

Zhao etal. (2002) 
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Table 2. Recommended Classroom-Level Instructional Technology Practices 


Prior Research Support for Practice 


Recommendation (1) 

Recommended 
by (2) 

Correlation with 
Technology Use (3) 

Oorrelation with 
Learning Outcomes 
(4) 

Controlled Studies 
on Learning 
Outcomes (5) 

Integration of technoiogy with 
learning goals and offline learn- 
ing activities 


Becker (1 994) 

Means & Olson (1 995) 

Wenglinsky (1 998) 


Technology used frequently 

Van Dusen & 
Worthen (1 995) 


Mann etal. (1998) 
Wenglinsky (1 998) 


Teacher present and facilitates 
learning when technology is used 


Sandholtz et al. (1 997) 


Powell et al. (2003) 

Teacher reviews software reports 

Efficient routines established for 
shifting in and out of technology 
use (classroom management) 

Coley et at 
(1997) 

OTA (1995) 


Powell et al. (2003) 

Low student-to-computer ratio in 
classroom 

Barnett (2002) 
Glennan & 
Melmed(1996) 
OTA (1995) 

O’Dwryeretal. (2004, 
2005) 

Oavalier & Klein 
(1998) 



learning outcomes. Most educators will expend the effort needed to inte- 
grate technology into instruction when, and only when, they are convinced 
that there will be significant payoff's in terms of student learning outcomes. 
Hence, to make technology an agent of education change, the field needs to 
understand the kinds of learning outcomes that technology can enhance and 
the circumstances under which that enhancement will be realized in prac- 
tice. Sound guidance on how to implement technology in ways that produce 
student learning gains is integral to efforts to use technology as a lever for 
education change. 

As illustrated in Tables 1 and 2, an extensive literature on “best practices” 
in technology implementation does exist. The first column in Table 1 lists 
common recommendations for school-level practices in support of instruc- 
tional uses of technology. 

The first column of Table 2 lists commonly recommended teachers’ class- 
room practices with respect to technology implementation. 

These tables also show that, in most cases, the basis for recommending 
the implementation practices is expert opinion or a correlation between 
the practice and the observed extent of technology use. Only a handful of 
articles document a correlation between an implementation practice and 
student learning outcomes. Very few studies with a rigorous, controlled 
design have examined the effects of one of the recommended technology 
implementation practices on student learning outcomes. A formal search of 
the ERIC and Psychinfo databases to identify empirical studies using a con- 
trol group design (either experimental or quasi- experimental) was conducted 
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in support of a large research study (Dynarski et al., 2007) sponsored by 
the Institute of Education Sciences. Only a single published study meeting 
these criteria (Powell, Aeby, & Carpenter- Aeby, 2003) was identified through 
this search.' Powell, Aeby, and Carpenter- Aeby (2003) found that teacher 
presence during use of instructional software and teacher review of software 
reports of student performance on the software produced greater student 
learning. Hence, we are urging schools and teachers to implement technol- 
ogy with little or no empirically based guidance on how to do so in ways that 
enhance student learning. 

An implication of the discussion above is that technology implementation 
practices need to be investigated in conjunction with studies of technology 
effects on student learning. Unfortunately, few large-scale studies have mea- 
sured both effects of technology on student learning and technology imple- 
mentation practices. A prominent exception is the congressionally mandated 
national experiment on the Effectiveness of Educational Technology Inter- 
ventions (EETI), which examined the effects of reading software for students 
in grades 1 and 4 and of mathematics software for students in grade 6 and 
algebra classes (Dynarski et al., 2007). EETI found that, on average, the effect 
size for using reading or mathematics software was not statistically differ- 
ent from 0 at any of the four grade levels included in the study. Within each 
grade level and product, the classes using the software did better than those 
that did not at some schools, whereas the classes using their conventional ap- 
proaches did better than those using the software at other schools. The only 
significant relationships between effect sizes and software implementation 
variables found in this study were larger effects in classes with more stu- 
dents per computer in grade 1 (contrary to a common recommendation for 
technology implementation) and a relationship between effect size and the 
amount of time students spent using the reading software in fourth grade 
(Dynarski et al., 2007). 

In contrast, a study of a large urban district’s implementation of the Wa- 
terford early reading software by Hansen, Llosa, and Slayton (2004) found 
that the amount of time students spent with the software was not correlated 
with measures of student learning. A randomized control trial of Acceler- 
ated Reader conducted by Nunnery, Ross, and McDonald (2006) found 
no relationship between the study’s quality of implementation index and 
student achievement growth. In short, despite the existence and extensive 
dissemination of conventional wisdom concerning how technology should 
be implemented, the evidence base for recommending particular practices is 
neither deep nor internally consistent. 

The research reported here was conducted with a subset of the EETI 
school sample to provide insights for those responsible for implementing 


1 Subsequent work with the technology implementation research uncovered a quasi-experimental study (Cole, Simkins, & Penuel, 2002) that 
found student learning benefits associated with teachers’ receipt of support from school-based technology integration specialists skilled in 
the design of project-based learning activities involving student use of multimedia technology. 
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reading and mathematics software by providing a closer look at school and 
classroom implementation practices. This study contrasts practices in schools 
whose students had above-average achievement gains in their first year of 
software use as part of the EETI study with those of schools where treatment 
classes had below-average gains. This correlational analysis used implementa- 
tion data from the EETI study as well as data from a set of follow-up inter- 
views and observations conducted with staff at 13 schools continuing to use 
the software they had implemented the prior year as part of the EETI study. 
This study focused on two central questions: 

• What classroom-level practices are associated with higher achieve- 
ment gains in classrooms using reading or math software? 

• What school-level practices are associated with higher achieve- 
ment gains in classrooms using reading or math software? 

To explore issues of software implementation, analysts identified those 
EETI schools where software-using teachers’ students experienced above-av- 
erage achievement gains and those whose students had below-average gains 
in the first year of the EETI software effectiveness study.^ From these two 
school subsamples, 14 schools were selected for follow-up — 7 in the above- 
average group and 7 in the below-average group. The 14 selected schools 
were using seven different software products (four reading products and 
three mathematics products) and included an above-average- and a below- 
average-gain school for each product. For each product, researchers looked 
for a high-gain school with a positive effect size and above-average use of 
the software for which a low-gain school matched on student demographic 
variables could be identified. For each product, schools were selected to be 
as similar as possible except for their differing levels of sfudent gains. 

The 14 schools selecfed for case sfudy were contacfed in April 2006 fo 
ascertain whether they would be willing to participate in this follow-up data 
collection by completing phone interviews or hosting a site visit. All of the 
schools initially agreed to participate, but one of the low-gain schools subse- 
quently dropped out of the data collection, resulting in a follow-up sample 
of 13 schools, as shown in Table 3 (p. 290). 

By virtue of the selection process, the two groups of schools differed in 
average class sfandardized achievement gain (0.77 for the high-gain group 
versus -0.70 for the low-gain group). As intended, they were very similar in 
terms of variables related to their staff and sfudent populations. The propor- 
tions of students eligible for free or reduced-price lunch, for example, were 
57% and 56% in high- and low-gain schools, respectively. 

The schools in the case study sample were using seven software products — 
four reading products and three mathematics products. Table 4 (p. 291) shows 

2 Identification of the sehools for case studies was based on information made available from the Effectiveness of Educational Technology 
Interventions (EETI) study (Dynarski etal,, 2007). 
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Table 3. Characteristics of High- and Low-Gain Schoois in the Foliow-Up Sample 


Variable 

High-Gain Sehoois 
(n-7) 

Low-Gain Schoois 
(n = 6) 

Teacher experience ievei (years) 

8.8 

12.7 

Teacher certification (percent) 

79 

83 

Urban scboois (percent) 

71 

50 

Free/reduceri-price iunch (percent) 

57 

56 

African American (percent) 

32 

36 

Hispanic (percent) 

16 

31 

Speciai education (percent) 

10* 

3 

Student-to-teacher ratio 

18.0 

16.0 

Pretest score (standardized) 

-0.14 

0.21 

Gain score 

0.77* 

-0.70 


* Significant at p < .05. 


the number of classrooms using each product and the instructional features of 
those products, as judged by instructional design experts on the research team.^ 

Method 

One pair of schools (a high- and a low-gain school both using the same 
product) at each grade level was designated for a site visit, which would 
involve interviews with the principal or other school leader and the school 
technology coordinator (if there was one), as well as with each teacher who 
had participated in the treatment condition in the EETI study. Site visits 
also involved observing each teacher twice — once while using the software 
with students and once while teaching the relevant subject (math or read- 
ing) without the software.^ For follow-up schools that did not receive a site 
visit, researchers conducted phone interviews with the principal, technology 
coordinator, and teachers using the same interview protocols employed on 
the site visits. They used the same interview protocols for high-gain and low- 
gain schools, and site visitors and did not inform interviewers of the school’s 
categorization as high or low gain. 

Analysts blind to the level of gains a school or teacher had experienced 
during their first year of software use coded the data obtained through 
interviews and observations for descriptions of school practices (such as 
principal support), classroom practices (actions undertaken by individual 
teachers), conditions (demographic variables and other characteristics 
existing prior to software implementation), and perceived outcomes. Data 
coding began with two analysts independently coding each paragraph of 


3 The coding team developed a set of instruotional features, such as incorporation of practice opportunities, on which all software products 
could be judged. Two coders independently reviewed products, retaining feature categories for which intercoder agreement was 80 percent 
or better. 

4 In some cases, this protocol had to be modified for elementary reading because the implementation model for the product was to have a 
portion of the students working independently on computers, whereas another portion worked with the teacher in a small group during all 
reading instruction, 
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the data forms for two schools. Interrater agreement for the independent 
coding was greater than 75%. A single analyst conducted the remaining 
coding. The coded data was entered into a qualitative analysis software 
database (ATLAS.ti) to facilitate identification of examples of particular 
practices and analysis of differences between high- and low-gain schools 
in terms of both teachers’ classroom practices and schoolwide supports for 
software implementation. 


Results 

The differences the analysts identified between high- and low-gain schools 
are reported below for teachers’ classroom practices as they use the software 
and for schoolwide supports for software implementation. 

Teacher Implementation Practices 

Level of software use. Teachers participating in the EETI software effective- 
ness study received training on use of the software, which included specifica- 
tion of the amount of time they should give students on the software each 
week. Software vendors’ recommendations for weekly use of their products 
ranged from 75 to 135 minutes. When each teacher’s reported use was com- 
pared to the usage recommended for the product in that class, the propor- 
tion of teachers meeting or exceeding vendor usage specifications in high- 
gain schools, at 64%, was not significantly different from that in low-gain 
schools (50%). The average weekly number of minutes teachers reported in 
high- and low-gain schools was roughly equivalent (119 and 102 minutes, 
respectively). Teacher reports indicated that the great majority of teachers 
were making a good-faith effort to have their students spend a significant 
amount of time with the software, and thus it is possible that level of use 
would be more strongly associated with achievement in implementations 
where usage levels varied more widely. 

Although the amount of time that teachers reported having their stu- 
dents use the software was not associated with student gains in the case 
study sample (or in three of four grades for the EETI sample as a whole), 
there was a significant relationship between student gains and the point in 
the school year when classes started software use. On average, teachers in 
the high-gain case study schools started software implementation 4.5 weeks 
after school started, whereas teachers in low-gain schools did not begin until 
7.7 weeks into the school year. The later start in low-gain schools did not 
appear to decrease the total number of hours the average student received on 
the software, as logged by the six software products from which such record 
could be obtained. The average annual software exposure was 23.1 hours for 
students in high-gain schools and 23.3 for students in low-gain schools. It 
may be that the speed with which a school ramped up for software imple- 
mentation was influenced by other factors that can also influence technology 
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implementation, such as the quality of the school’s technology infrastructure 
and support or the school’s overall management efficiency. 

Classroom management. In interviews, teachers talked about the need to 
develop classroom routines for moving onto and off the software. A num- 
ber of the teachers said that students needed to learn how to execute this 
transition and log on and off the software independently for the class to run 
smoothly. When asked what advice they would give another teacher using 
the software for the first time, teachers were more likely to provide recom- 
mendations on classroom management than on any other topic. 

For those teachers who were observed in the act of teaching, all of the 
teachers in high-gain schools were rated as effective in terms of classroom 
management compared to just 17% of observed teachers in low-gain schools.^ 
Thus, effective classroom management routines for software use appear to be 
important. In observed classrooms, effective classroom management appeared 
to allow greater focus on the instructional activities of the software rather than 
the logistics of its use. Ironically, this aspect of implementation is seldom men- 
tioned in the literature on educational technology “best practices.” 

Facilitation during software use. Software vendors recommend that the 
regular classroom teacher be present while their students use instructional 
software. In addition to managing classroom behavior and activity flow, 
feachers also provide more or less subsfantive support for students’ learn- 
ing during periods when they are using the software. Software products are 
typically designed so that students can use them independently, but technol- 
ogy advocates often make the case that by engaging all students in learning 
independently, the software provides the opportunity for teachers to interact 
one on one with those students needing the most help. 

Nearly all of the case study teachers (25 of 27) reported being present in 
the room while their students used the software. Observed teachers varied, 
however, in the amount of substantive support they provided for students’ 
learning during software use. Some of the case study teachers were observed 
rotating through the room of software-using students, identifying students 
who were experiencing difficulty and working with them one on one. For 
example, in one sixth grade mathematics class using software in a com- 
puter lab, the teacher circulated around the room, interacting not only with 
students who raised their hands but also with students she observed to be 
progressing slowly (Means et al, 2006): 

The teacher helped one student who asked, “When you have a -f and - 
sign, which do you keep?” 

“Think about what the sign needs to be to get you to -f2x,” the teacher said 
as she directed the student’s attention to what he had written on a piece of 


5 Classroom management quality was judged on the basis ot the proportion of time the class spent in learning and Instruotlen, as opposed te 
moving around, preparing materials, or student dlsruptiens. 
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scrap paper. “Think about that. That gets you to -2x + x. . ..There you go, 
look at that! Good.” 

The student responded, “Right, right, thank you,” and the teacher moved 
on to another student who had looked up for help. 

In a sixth grade mathematics classroom at another school, the teacher 
was in the room as students worked with the software but spent about half of 
the class period on activities such as grading homework and talking on the 
phone that did not support students’ work with the software. 

The observed quality of teacher facilitation was measured for only the six 
teachers in the follow-up sample who could be observed teaching with the 
software during the school site visits, and two thirds of the observed teachers 
in both high- and low-gain schools engaged in facilitation of learning in the 
software sessions the researchers watched. 

Articulation and integration of instruction with and without software. Coor- 
dination of learning done through the software with instruction in the same 
topic that does not involve software emerged as an issue in interviews with 
the case-study teachers. None of the classes did all of their reading or math- 
ematics learning through software. Five of the seven software products in 
this study were designed as supplements to regular instruction rather than 
as the core reading or mathematics curriculum. Even in classrooms using a 
mathematics product intended as the core curriculum, the product included 
offline as well as technology-based activities and materials. For products 
intended as supplements, the software was not a component of the core cur- 
riculum, and therefore teachers were using software produced by one vendor 
and a textbook or other set of core-curriculum instructional materials from 
another source. This situation raised several challenges for the teacher. First, 
the teacher must coordinate the topics across the two sets of instructional 
materials. Most of the software products allow teachers flexibility in sequenc- 
ing software modules so that the use of software on a given topic or skill can 
be fit into a logical place in the core curriculum. Although quite feasible, this 
effort does take time. Teachers at some of the case-study schools reported 
that they had not worked through the software themselves or compared its 
coverage to that of their core curriculum to identify areas of overlap prior to 
implementing the software with their students. In some cases, teachers were 
aware that they could change the sequence of the software modules to match 
what they were doing in class, but had not done so. As one teacher said, “A lot 
of times we just let it [the software] roll the kids into the next level.” 

Other teachers did take the time to become familiar with the software and 
think about how to articulate the software modules with classroom activi- 
ties. A high school algebra teacher, for example, chose the software modules 
her students would use each week based on what she would be teaching in 
the classroom. In addition to instances of adapting the sequence of software 
activities to match what was happening in the core curriculum, there were 
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also instances of teachers adapting what they did in class on the basis of 
insights gained from observing students working with the software. 

This observation from a first grade class implementing reading software 
illustrates the way in which students’ software-based activity can influence a 
teacher’s core instruction (Means et al, 2006): 

The first grade teacher began by reviewing the vowels. In her interview, 
the teacher said she did this because she had heard one of the students 
singing an “a-e-i-o-u” song from the software during lunch. She said that 
she often listened to students humming to themselves or singing songs 
from the software and used that as an indication of where they were in the 
software and where their skills might need reinforcing. 

A second issue related to integrating software use and other instruction 
is the way that concepts or procedures are presented. This appeared to be 
particularly troublesome in mathematics, when different terminology and 
different procedures for handling problems of the same type were likely 
to be conveyed in the textbook and in the software. Mathematics teachers 
at one school found this discrepancy to be so confusing to their students 
during the first year of using the software that they decided not to teach the 
same topic with the textbook and the software in close temporal proximity 
in the second year, reasoning that students would be less confused by the 
representation of something in the software if they had had time to forget 
how it was presented in the text. 

Analysts coded teachers’ observed practices as their students were us- 
ing software (where available) for evidence that the teachers were helping 
students integrate ideas and representations covered by instruction with and 
without software so that students could make connections between what 
they learned through the two modalities.'^ A high school algebra class pro- 
vided an example of a teacher bringing content from the software into her 
classroom instruction (Means et al., 2006): 

The high school algebra class returned from the computer lab and sat 
down in rows of desk chairs facing the blackboard. At 2:55 p.m. the class 
began with the teacher saying, “Some of you were looking at this in the 
computer lab.” She then began talking about simplifying rational expres- 
sions and went through some problems on the board. 

We also observed examples of teachers referring to what the students 
had experienced in class as the students were working with software in the 
computer lab, as illustrated by the observation below from a middle school 
mathematics class (Means et al, 2006): 


6 Unless there was evidence of a lack of implementation of nonsoftware components, those teachers who used a eore curriculum product 
with both computer-based and non-computer-based material were coded as having integrated software use and other instruction. 
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As students worked with the math software in the computer lab, their 
teacher looked for students with raised hands or puzzled looks and pro- 
vided individual assistance. The students were working with problems 
calling for them to do arithmetic operations with combinations of posi- 
tive and negative integers. After noting that several of the students were 
struggling, the teacher addressed the class as a whole: “Remember when 
you add integers, use those algebra tiles we talked about in class. Red was 
for positive numbers and blue for negative numbers.” (The software uses 
a number line depiction rather than algebra tiles, but it also uses red for 
negative numbers and blue for positive numbers.) 

There were too few observations of software use during the follow-up 
study to support firm conclusions, but teachers’ responses to interview items 
about integration of software-based and other instruction did suggest a 
relationship between integration and gains. Eighty-six percent of teachers in 
high-gain schools said that they had done this kind of integration, compared 
to 77% of teachers in low-gain schools. 

Those teachers who actively facilitate their students’ work with software 
are in a position to adapt their non-technology-based instruction by draw- 
ing on insights gained from interactions with the software. The software in 
essence provides the teacher with formative assessments, which research 
suggests can improve learning if the teacher uses them to guide future in- 
struction (Black & Wiliam, 1998). Some of the case study teachers described 
this kind of practice. An algebra teacher, for example, reported that she finds 
that helping students individually while they are using the software is a good 
way of gauging their understanding and identifying areas to reteach. She 
said that students “who won’t ask questions in the classroom will ask in the 
computer lab because they don’t feel so much on the spot.” She described her 
practice of taking the questions that students asked in the computer lab back 
into the classroom and reteaching concepts as needed. 

Use of software data to inform instruction. Software vendors have long 
urged teachers to run the automated reports provided by their products 
and review them on a regular basis. Typically, these reports provide data on 
individual student progress through and mastery of the various software 
skill modules, as well as summary-level information for the class as a whole. 
Although the vendors’ recommendation is rooted in concerns over teach- 
ers’ support for software use and the desire to make sure the software is used 
appropriately, this practice appears to have broader benefit. Teachers at the 
high-gain schools in the follow-up sample reported doing frequent review of 
the student performance reports the software generates. Seventy-eight per- 
cent of the teachers in high-gain schools said they looked at software reports 
for all their students once a week or more, compared to 17% of teachers in 
low-gain schools. Thus, use of software-generated data reports was one of 
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the largest differences between high-gain and low-gain implementations. 
Although it may be that achievement- oriented teachers implement many 
useful practices in addition to looking at data, our qualitative data are con- 
sistent with the hypothesis that the use of student performance information 
generated by software products helps teachers target their instruction to the 
things that students need to learn. Two examples illustrate teachers’ use of 
the detailed student progress reports generated by software systems (Means 
et al, 2006): 

A fourth grade teacher reported that she looked at the software student 
performance reports on a monthly basis to monitor student progress 
and check what students were working on. She used the results of one 
report to group students during the core reading instruction time by topic 
mastery, allowing her to differentiate the pacing of her core instruction. 
“These kids are working on similes; these kids are working on context 
clues. I tried to coordinate it,” she said. She also used software reports dur- 
ing parent- teacher conferences to emphasize or clarify a point she wanted 
to make with parents about their child’s progress. 

A teacher at another elementary school said she looked at the software 
reports often and used the data to decide how to group students in her 
class. She said she also looked at the software “to help with words for class 
and give students individualized spelling words based on where they are 
in the software.” The reports helped her decide what to teach, based on 
where students were in the software and how they were doing. “I love us- 
ing those reports and being able to see exactly where kids are,” she said. 

“I can sit down with a parent in a conference, and there’s no guessing on 
my part.” 

Developing a system to motivate software use. Some of the software ven- 
dors recommend instituting a motivational system around software use — 
creating a visible chart showing modules completed, giving certificates for 
accomplishments on the software, or using software performance in grad- 
ing. Some of the individual teachers in both the high-gain and the low-gain 
schools described setting up such systems, usually in the form of a public 
chart showing each student’s record of software module completion. The dif- 
ference between teachers in high-gain schools and those in low-gain schools 
in use of this technique was not significant. Moreover, when the correlation 
between this technique and gain score is computed for all EETI treatment 
classrooms, a negative relationship between this practice and student gains is 
found (perhaps because teachers whose students are not progressing resort 
to extrinsic rewards). 

Table 5 (p. 298) summarizes the differences befween teachers in high- and 
low-gain schools in terms of teacher-level software implementation practices. 
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Table 5. Teacher Implementation Practices in High- and Low-Gain Schools 


Practice 

Teachers in High-Gain Schools 

Teachers in Low-Gain Schools 

Teacher-Reported Practices a 


Met vendor software weekly usage guideline 

64% 

50% 

Reviewed software reports for all students weekly 

78% 

17% 

Integrated software use with other instruction 

86% 

77% 

Instituted motivational system 

64% 

46% 


Observed Practices b 


Managed classroom effectively 

100% 

17% 

Facilitated learning during software use 

67% 

67% 


a Number of teachers responding to interview item varied between 1 2 and 1 4 for the two sets of schools, 
b Number of teachers observed and scored on a variable varied between 3 and 6 for the two sets of schools. 


School-Level Implementation Practices 

In addition to teacher behaviors and choices that teachers make about how 
to implement software with their students, the school as a whole provides 
supports for software implementation. The interviews that researchers 
conducted with school staff participating in the follow-up study addressed 
some broader school implementation issues that had not been covered in 
the EETI data collection. These include integration of technology use with 
a schoolwide instructional vision, principal support for use of the software, 
and teacher collaboration around software use. Additional schoolwide prac- 
tices related to software implementation that were documented as part of the 
effectiveness study include the technology infrastructure, technical support, 
and receipt of additional formal training on software use or implementation. 
High- and low-gain schools were contrasted in terms of all of these school- 
wide practices. 

Consistent instructional vision. In schools with a consistent instructional 
vision, the principal and the treatment teachers expressed similar coherent 
views of how the subject (reading or mathematics) should be taught and the 
role that the software should play in implementing that instructional vision. 
Such consistency can be illustrated by reports from the staff of a large middle 
school serving a low-income student body that included many students who 
were not yet fluent in English (Means et al., 2006): 

All of the interviewed staff indicated that their top priority was to achieve 
the state/district content standards on the schedule designated in the 
district’s instructional guide and pacing chart. The principal and both 
software-using teachers noted in separate interviews that many of their 
students did not come in with the skills that the district’s instructional 
guide assumed and that, although they were supposed to be teaching the 
more advanced skills in the district instructional guide, they also needed 
to work on basic skills for students who had not yet mastered them. They 
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all cited the software as useful for this purpose. They thought it would 
be better if students acquired basic skills before being moved to more 
advanced mathematics, but had concluded that they had to teach both 
together and saw the software as a useful tool for doing so. 

The majority of high-gain schools (four of seven) were coded as having 
a consistent instructional vision. In contrast, only two of six low-gain 
schools were judged to exhibit this kind of consistency. Lack of a consis- 
tent view of the software’s role was illustrated by the staff in a low-gain 
school, as described below (Means et al, 2006); 

The principal had little involvement with the study and was unsure how 
well the software fit with current schoolwide initiatives. One of the two teach- 
ers using the software taught low-skilled and English-language-learner stu- 
dents and said that the software would be good for honors students but not for 
his students because the language and concepts were too advanced. The other 
software-using teacher taught honors students and said that they got bored 
with it, but he expected that the software would be good for remediation. 

Principal support. In addition to being instrumental in forging a consistent 
instructional vision, principals can provide support for use of instructional 
software, not just in the form of permission and supportive verbal state- 
ments but also in terms of concrete actions, such as giving the classes using 
the software priority access to computer resources and arranging for joint 
planning periods or other paid opportunities for teachers to gain proficiency 
with the software and to plan for its use. The majority of high-gain schools 
in the follow-up sample (five of seven) had principals who supported the 
software implementation in those ways; the majority of low-gain schools 
(four of six) did not. 

Teacher collaboration. Beyond the support from the principal, support 
from one’s colleagues appears to be another factor present in schools that 
achieve learning gains with technology. All seven of the high-gain schools 
that could be scored for this variable reported that their teachers collaborated 
and supported each other on use of the software product. Only one third of 
the low-gain schools (two of six) reported this kind of teacher collaboration. 

Two teachers at an elementary school implementing the reading software 
illustrated the kind of teacher collaboration found in the high-gain schools 
(Means et al, 2006); 

Collaboration between the two teachers using the software at this elemen- 
tary school was very close. The younger teacher, who was in her second 
year of teaching, “handles the technical side, does reports, registers kids.” 
The older, veteran teacher mentored her younger colleague on instruc- 
tional strategies for elementary reading. They planned together at the 
beginning of the year and got together throughout the year to look at 
software reports and discuss student progress. One of the teachers noted 
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Table 6. Proportion of Schools with Schoolwide Practices and Supports, by Gain Status 


Practice or Support 

High-Gain Schools (n = 7) 

Low-Gain Schools(n = 6) 

Consistent instructional vision 

4 of 7 

2 of 6 

Principal support for software use 

5 of 7 

2 of 6 

Teacher collaboration around software use 

7 of 7 

2 of 6 

Satisfactory onsite technical support 

6 of 7 

3 of 6 

Receipt of addifional formal framing 

3 of 7 

1 of6 

Receipf of informal face-to-face support 

5 of 7 

3 of 6 

Access to help desk, e-mail, website 

6 of 7 

5 of 6 


that they even “get in competition a little with the scores.” The schools 
technology coordinator commented that “they work well together; they’re 
both committed to the program.” 

A number of researchers have suggested that the presence of strong ties 
among teachers who view themselves as a learning community will facili- 
tate the adoption of technology (Frank, Zhao, & Borman, 2004; Strudler & 
Hearrington, 2008). In these case studies, however, there was no difference 
between the proportions of high- and low-gain schools in the follow-up 
sample that were judged to emphasize teacher collaboration generally (five 
of seven high-gain schools and four of six low-gain schools described them- 
selves as collaborative). This finding suggesfs fhat fhe focus of fhe collabora- 
tion, not just a generally supportive climate, is important. 

Technology infrastructure and technical support. In implementing the EETI 
effectiveness study, the research team took steps to make sure that each 
school had the technology infrastructure needed to implement the software 
assigned to treatment classrooms. The study team worked with districts 
to identify hardware and software needs such as computers, headphones, 
memory, and operating system upgrades, and the study purchased the up- 
grades as needed. Thus, all of the schools in the case studies (as well as those 
in the larger EETI study) had technology infrastructures that were, at least in 
theory, adequate for running the assigned reading or mathematics software. 

When we asked case study teachers about the technology support avail- 
able to them, we found that high-gain schools nearly always had on-site 
technical support that teachers considered good (six of seven high-gain 
schools). Half of the case study low-gain schools (three of six) were similarly 
happy with their on-site support while half were not. Hence, the qualitative 
data suggest that the quality of on-site technology support, rather than its 
mere presence, is important and that good local support is not sufficient but 
may be necessary to ensure positive outcomes with technology. 

Receipt of additional training and support. In a typical implementation of 
commercial software, the school or district purchasing the software has the 
option of obtaining training and support services from the vendor. In the 
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EETI study, these services were provided by vendors as part of the agreement 
around their participation. All of the commercial vendors with a product in 
the study conducted formal initial training for the teachers who would be 
implementing their software. In addition to this formal training, vendors 
were permitted to provide additional formal training sessions during the 
school year, face-to-face informal support as part of school visits, or assis- 
tance through a help desk, email or a web site. Somewhat larger proportions 
of the high-gain schools received additional formal training and face-to-face 
informal support from the software vendors, but the differences were not dra- 
matic and the samples are small. A similarly large majority of teachers in both 
high- and low-gain schools reported having access to technical assistance in 
the form of a help desk, e-mail, or website supporting use of the software. 

Table 6 summarizes the differences between high- and low-gain schools 
in terms of schoolwide implementation practices. 

Changes in Implementation Practices from Year 1 to Year 2 

Many changes in implementation practices in teachers’ second year with the 
software appeared to reflect teachers’ increased knowledge and confidence 
around software use. Nearly half of the teachers reported that technology 
problems were reduced in their second year of software implementation. 
Teachers at six of the 13 follow-up schools reported fewer technology prob- 
lems in Year 2. One of these schools had gained a technology coordinator, 
and two schools that had had major start-up delays in Year 1 did not have 
this same problem in their second year of implementation. Teachers also 
credited their own increased familiarity with technology and with the pro- 
grams for a decrease in technology problems. Technology problems do not 
always disappear with experience, however; two schools said that they had 
had more rather than fewer technology problems in Year 2. In both cases, a 
change in the school’s technology infrastructure required reconfiguration of 
the technology for running the software. 

One might expect teachers’ increasing comfort with the technology to 
lead to more software use in their second year of implementation, but this 
was not the case. Nine teachers from five different schools reported us- 
ing the software for less time in Year 2 than in Year 1, and only one teacher 
reported increased use. Time pressures and competition with other school 
improvement initiatives were the most frequently cited reasons for reduc- 
ing the amount of time devoted to software use during the second year of 
implementation. Three of the schools where there was less software use in 
the second year had introduced a new core curriculum in the area covered 
by the software. Teachers said that the new curriculum required adjustment 
on their part and integrating the software with the new curriculum was dif- 
ficult. At one of these schools, for example, teachers felt the new curriculum 
required them to do more whole-class instruction, making it more difficult 
to find time for software use. 
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Discussion 

The findings from this study generally are consistent with the usual rec- 
ommendations for school-level supports for technology in the education 
literature but are less similar to common recommendations for practices 
within classrooms implementing technology. The four school-level practices 
for which the study found support were; 

• Establishment of a consistent instructional vision 

• Principal support for software use 

• Teacher collaboration around software use and 

• Satisfactory on-site technical support. 

These recommendations are among those commonly made in the educa- 
tional technology literature (as shown in Table 1, p. 286). 

The classroom-level practices receiving the strongest empirical support 
from this study were: 

• Reviewing software reports for all students weekly and 

• Managing the classroom eifectively. 

Similar practices can be found in Table 2 (p. 287; “Teacher reviews soft- 
ware reports” and “Efficient routines established for shifting in and out of 
technology use”), but there are fewer citations for them in the literature than 
there are for other practices, such as a low student-to-computer ratio, that 
did not receive empirical support either in the qualitative study of 27 class- 
rooms reported here or in the large EETI study involving 132 classrooms. 

One implication of this study for classroom practice and software im- 
plementation efforts is that teachers should be urged to capitalize on the 
assessment data that instructional software makes available. Most instruc- 
tional software will adapt the learner’s experience to the results of embedded 
assessments automatically. But teachers can use the software assessment 
reports also to identify specific areas where they can do more to support 
students during their regular classroom instruction, as was described by a 
number of case study teachers. Software reports can also bring to light indi- 
vidual students’ motivational issues or learning disabilities that might have 
gone unnoticed during whole-class instruction. 

A second implication is that training and support around instructional soft- 
ware should pay more attention to the details of classroom management. Teach- 
ers work with classes of different sizes, in different physical settings, and with 
different kinds of students. One of the teachers in the EETI study, for example, 
taught pre-Algebra to a large class of students in a space intended as a lecture hall. 
Students were widely spaced out in seats with small flip-up arms designed to hold 
a pad of paper. Distributing laptop computers and juggling papers and computers 
at the same time under such conditions is a challenge. Teachers also have differ- 
ent habitual routines for organizing and orchestrating class activities. All of these 
factors can influence the ease with which students can be transitioned into and 
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out of technology-based activities. As with professional development on other 
aspects of teaching, it would be helpful for teachers to be able to observe examples 
of efficient routines for transitioning in and out of software use within the kind of 
setting in which they teach, and then to try out their transition strategy in their 
own classroom and school, with the opportunity to receive feedback and sugges- 
tions for improvement from experienced software-using coaches. 

The study has implications also for future research. It raises but does not 
resolve questions about how teachers can best facilitate their students’ work 
as students are engaging with software designed for independent individual 
use. The literature on classroom implementation of technology highlights 
the role of the teacher as a facilitator of students’ technology use. This facili- 
tative role is usually described as a move toward student- centered instruc- 
tion and away from the more typical teacher role of directing classroom 
activities. However, other than giving students more responsibility, there is 
actually very little guidance in the literature about the most effective teacher 
facilitation practices when students are working with software. In these case 
studies, researchers categorized an identical percentage (67%) of observed 
teachers in high- and low-gain schools as “facilitating” students’ use of soft- 
ware. It would be premature to suggest that teachers’ facilitative behaviors do 
not affect student-learning gains, but this null result does suggest a need for 
a more precisely defined concept of teacher facilitation during software use. 
In interviews, teachers report gaining insights into their students’ think- 
ing by observing or working with students who are engaged with instruc- 
tional software. But how should the teacher interact with the student to best 
support his or her learning with the software? When should teachers give 
answers and when should they give hints to students struggling with the 
online content? Should teachers attend primarily to motivational and be- 
havioral issues or treat their students’ software use as a “teachable moment” 
for individual students? Establishing an empirical basis for making recom- 
mendations in these areas requires theory development around facilitation, 
the development of reliable measures of different aspects of facilitation, and 
a program of systematic research. 

Given the relatively small number of case study schools and of software- 
implementing teachers within those schools, findings from the present study 
should be interpreted with caution. Certainly failure to find significant differ- 
ences between high-and low-gain schools in terms of an often-recommended 
implementation practice could be attributable to limited statistical power. But 
those implementation variables for which significant differences did emerge, 
especially in cases where the case study data and the larger EETI data set pro- 
vide converging evidence, merit attention. Here we have evidence that the prac- 
tice is associated with learning gains, not just more frequent use of technology. 

The descriptive data presented in the appendixes also highlight larger is- 
sues having to do with the way teachers juggle the requirements of software 
implementation with everything else they are trying to do in the classroom. 
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Case study interviews and observations point to the need for more contex- 
tualized studies and a broader view of technology implementation. Rather 
than treating technology as a “thing” which is present or absent, researchers 
and educators need to look at instructional activity systems in context. Any 
piece of hardware or software is but a part of the instruction that students 
are exposed to in a given domain. The importance of teacher management 
of transitions onto and otf of software, teachers’ articulation of connec- 
tions between online and offline instruction, and teachers’ struggles to fit 
technology-based learning activities into schedules dominated by core 
curricula that are not technology-based underline the importance of taking 
this broader view. 

Building on the concept of an instructional regime, as described by 
Cohen and Ball (1999), educators and policymakers need to stop thinking 
of learning software as an intervention in and of itself and to think instead 
of broader instructional activity systems (Roschelle, Knudsen, & Hegedus, 
2009) defined by: 

• Learning content 

• Learning activities, only some of which are technology -based 

• Articulation between a given learning activity sys- 
tem and other systems the student is exposed to 

• Teacher professional development and collaboration around 
implementation of the instructional activity system 

• Assessments for learning 

• Use of data to refine the instructional activity on an ongoing basis. 

Such a framework has obvious implications for intervention design and 
what may be less-obvious implications for research. We need to get beyond 
the simplistic “what works” view of learning technology as a “pill” that will 
have the same effect on outcomes for anyone in any context. We have ample 
evidence that implementation practices matter, and the instructional activity 
system framework suggests basic categories for the kinds of data that should 
be collected in any study of a technology-supported intervention in action. 

The learning technology field needs implementation research contrasting 
the student outcomes produced by different implementation strategies and 
practices for the same intervention, not just comparisons of treatment versus 
no-treatment conditions. Education policymakers and practitioners need to 
think about the implementation of technology-supported interventions as a 
process of iteration and refinement rather than as a pill to be selected on the 
basis of “what works” evidence. Partnerships between the teachers, schools, 
and districts implementing technology-supported reforms and those who 
design and research such reforms can support a process of collaborative 
elaboration and adaptation of the intervention to fit local circumstances, fol- 
lowed by a cycle of implementation with monitoring of both the implementa- 
tion process and outcomes for students. Findings of the implementation cycle 
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can then inform another round of intervention refinement, again followed by 
implementation, monitoring, and improvement. Only by defining, measur- 
ing, and analyzing implementation variables and context along with student 
outcomes (Means & Penuel, 2005) can we gain the understanding that will 
support the implementation of technology-supported interventions in a way 
that optimizes student learning. 
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